Computers are often used to search and access information from large data bases. Commonly, text search engines are used to search and access text data bases using text queries and/or queries with logical operations on text. There are many systems in the prior art that perform these text searching functions. For example, the SMART system from Cornell and the InQuery engine from the University of Massachusetts at Amherst. See G. Salton and M. J. McGill "Introduction to Modern Information Retrieval" (McGraw-Hill, N.Y., 1983) and J. P. Callan, W. B. Croft, and S. M. Harding, "The INQUERY Retrieval System", Proceedings of the 3rd International Conference on Database and Expert Systems, September, 1992, which are herein incorporated by reference in their entirety.
Recently, computers have been used to store, search, and access multimedia documents from multimedia databases. Multimedia is information that can contain text, images, audio, video, and/or any other type of sensory information. A document (or multimedia document or electronic document) is one or more records of text and/or other multimedia information that is typically viewed at a workstation and/or stored in a multimedia database. The information on any of the records can have multimedia aspects, that is, one or more of the records can contain one or more multimedia types (text, images, video, animation, etc.)
Different types of search engines have been developed in the prior art to handle different types of content.
Images in multimedia documents in a multimedia database are searched and accessed using an image search engine. An image search engine works by first building a database in which a set of features are stored for each image that is indexed. In response to a query, which is expressed in terms of the features of the desired images, the image search engine searches the database for feature sets that most nearly match the query set. The result is a list of the corresponding images. Image search engines in the prior art include QBIC and PictureBook. See Niblack, W., Barber, R., Equitz, W., Flickner, M., Glasman, E., Petkovic, D., Yanker, P., Faloutsos, C., and Taubin, G., "The QBIC project: querying images by content using color, texture, and shape", Proceedings SPIE--International Society of Optical Engineering (USA), Volume 1908, 1993, pages 173-187. See also Pentland, Alexander P., Picard, Rosalind W., and Sclaroff, Stan, "Photobook: tools for content-based manipulation of image databases", Proceedings of SPIE--The International Society for Optical Engineering, Volume 2368, 1995, pages 37-50. Both of these references are herein incorporated by reference in their entirety.
In addition to text and image searching, the prior art contains search engines that search on parameters, also known as attributes. Parametric search engines generally function with tables of data, in which each row in the table represents an object and the columns represent parametric data associated with the object, such as its author or date. A parametric search engine returns a list of the rows which contain the combination of parameters specified in the query.
An example of a parametric search engine is the IBM DATABASE2 (or DB2) relational database system. (IBM and DATABASE 2 are trademarks of the International Business Machines Corporation.)
New technologies, like Digital Libraries, give users access to huge amounts of information, often in the form of multimedia documents, consisting of text, images, sound, and video clips. In many systems, each document can have associated with it, parametric data such as a document number, author, length, price, etc. Users who wish to find relevant documents need to be able to specify conditions on the content (e.g., that the text contains the word "Cadillac" and at least one image, e.g, a picture of a pink car) and/or on parametric data (for example, that the model year is earlier than 1960.)
Searching for particular information in technologies like Digital Libraries, especially over the Internet or the world wide web (WWW), is a formidable task. In general, the prior art allows searching for only one type of multimedia per search request. For example, a text search engine will search text parts which contain text that match a text query item like "Porsche". This text search engine will return a hit list of text parts that contain the words "Porsche." Alternatively, an image search engine will search for images that satisfy an image query item. For instance, an image query item can define shades of red. The image search engine will return hit list of images containing those shades. Also parametric search engines search parameters that describe entire documents and return a hit list of documents that satisfy the parameters in a query item.
Generally in the prior art, a user has to issue a separate query for each media type.
However in the prior art, image and parametric query searches are sometimes used together. In these cases, the parametric queries are used to filter out those documents that do not satisfy the parametric parameters in the query item. The remaining documents are returned in a "parametric hit list" and then a separate image search is done on these remaining documents. In these cases, the information in the documents that do satisfy the parametric search but do not satisfy the image search are lost. For example, a parametric search of "Porsches older than 1994" produces a hit list of Porsches made before that year. An image search defining shades of red, reduces that list to those Porches produced before 1994 that are red. All documents having information about Porsches older than 1994 but having a color other than red are lost.
Note that the prior art assumes that a logical "AND" operation is performed between the parametric and image searches. No other logical operation between the results of the parametric search and the results of the image search can be performed.